feat(server): lossless speculative tool execution — stream early dispatch, dependency waves, end-of-turn KV, prefetch-prefill (x1.44 on live-API agent loops) - #614

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/server/tool_speculation.cpp">

<violation number="1" location="server/src/server/tool_speculation.cpp:127">
P2: The child receives the entire server environment, including unrelated credentials and model configuration. Construct a minimal allowlisted environment instead of forwarding `environ`.</violation>

<violation number="2" location="server/src/server/tool_speculation.cpp:779">
P2: The FD-isolation guarantee is only enforced on glibc >= 2.34. `addclosefrom_np` is guarded by `__GLIBC_PREREQ(2, 34)`, so on glibc 2.17-2.33, musl-based Linux, or any non-glibc POSIX build the child executor inherits every open descriptor from the long-running server (listening sockets, live client connections, model IPC pipes, accelerator descriptors) — directly contradicting the header comment "The executor receives only stdin/stdout/stderr" and the PR's privacy claim. The failure is silent: the code compiles and launches without any FD isolation.</violation>

<violation number="3" location="server/src/server/tool_speculation.cpp:802">
P2: When CPU affinity is configured, `posix_spawn` lets the executor run before `pin_and_verify_child_cpu_affinity` applies the mask. Startup work can consume model CPUs before isolation; spawn through a pre-exec affinity handshake instead.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/benchmark_cpu_tool_speculation.py">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/benchmark_cpu_tool_speculation.py:334">
P1: When model generation consumes the configured timeout, `finish_executor` grants the tool another full timeout because `communicate` starts timing too late. Compute the remaining timeout from `handle["started"]` before waiting.</violation>

<violation number="2" location="optimizations/ooo_spec_lucebox5_cpu/benchmark_cpu_tool_speculation.py:336">
P2: Timeout and miss cancellation kill only the executor leader, so forked descendants can survive and retain the CPU lane. Launch the executor in its own process group and terminate the group on every cleanup path.</violation>

<violation number="3" location="optimizations/ooo_spec_lucebox5_cpu/benchmark_cpu_tool_speculation.py:504">
P2: If `post_model` raises after the private miss executor starts, `run_direct_miss` skips cleanup and the process continues using reserved CPUs. Put the model request and cleanup in a `try/finally`.</violation>

<violation number="4" location="optimizations/ooo_spec_lucebox5_cpu/benchmark_cpu_tool_speculation.py:565">
P2: The private_miss_result_hidden qualification gate is vacuous: run_direct_miss hardcodes 'private_result_exposed': False rather than deriving it, and the private executor is launched by the benchmark itself, so it is never routed through the engine and cannot expose a result. The check in qualify() therefore always passes and adds no real privacy coverage. The native() phase's miss_check ('result' in miss_metadata) is the actual isolation test. Either drop the qualify() gate or compute it from the native path; don't report it as a passed safety gate.</violation>

<violation number="5" location="optimizations/ooo_spec_lucebox5_cpu/benchmark_cpu_tool_speculation.py:1157">
P2: The automatic-Qwen production gate can run without explicitly verifying a qualified profile or disjoint model CPUs. Require the same profile and model/tool affinity checks as the `native` gate before measuring pairs.</violation>
</file>

<file name="server/src/common/qwen3_tool_predictor_ipc.h">

<violation number="1" location="server/src/common/qwen3_tool_predictor_ipc.h:29">
P2: When the configured IPC binary never sends its initial status, `start()` blocks in `BackendIpcProcess::start()` and server startup never reaches `run()`. Add a bounded readiness deadline, terminate the child on expiry, and treat the native predictor as unavailable so the configured HTTP fallback or normal server path can proceed.</violation>
</file>

<file name="server/src/server/native_semantic_tool_predictor.cpp">

<violation number="1" location="server/src/server/native_semantic_tool_predictor.cpp:52">
P2: When prompt construction or tokenization exceeds `config_.timeout_ms`, `predict()` still blocks before its only deadline check, so before-model requests exceed the configured timeout and HTTP fallback loses its remaining budget. Make preprocessing deadline-aware or otherwise bound it before entering the predictor lane.</violation>
</file>

<file name="server/src/server/server_main.cpp">

<violation number="1" location="server/src/server/server_main.cpp:626">
P2: When `--tool-hint-native-gpu` is malformed, `std::atoi` silently selects GPU 0. Parse this option strictly and reject non-numeric or out-of-range values before starting the native predictor.</violation>

<violation number="2" location="server/src/server/server_main.cpp:673">
P2: When `--tool-hint-execution-confidence` is malformed, `std::atof` converts it to `0` and startup accepts it. Parse the argument strictly and reject trailing or non-numeric input instead of silently changing the admission policy.</violation>
</file>

<file name="server/src/common/qwen3_tool_predictor_ipc_daemon.cpp">

<violation number="1" location="server/src/common/qwen3_tool_predictor_ipc_daemon.cpp:90">
P2: For predictions of at least three tokens, this lane produces later logits from a skipped Qwen3 KV-cache position. Fix `Qwen3Backend::do_decode`'s position update before routing predictor requests through this backend.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/results/trace-compiled-engine-qwen-production-6pairs-compact.json">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/results/trace-compiled-engine-qwen-production-6pairs-compact.json:653">
P2: The trace-compilation provenance recorded in this results artifact cannot be reproduced from the repo. `pattern.training_report` and `pattern.workflow_registry` point to absolute `/home/lucebox5/tool-spec-cpu-20260813/results/...` paths to files that are not committed (`multiturn-cached-wordref-production-6tasks.json`, `trace-workflow-registry.json`), while the README's reproduce command feeds `results/trace-compiled-training-traces.json`, whose sha256 is `ba73d1aef7c6...`, not the recorded `training_report_sha256` `2475697d...`. The recorded hashes therefore match nothing in the repo, so the 6/6 exact-hit and gate evidence cannot be independently regenerated or verified. Record the canonical source path (relative to the repo), commit the workflow registry that defines the `execute_customer_workflows` macro allowlist/canonicalization, and make the recorded sha256 correspond to the committed input.</violation>
</file>

<file name="server/src/server/http_server.cpp">

<violation number="1" location="server/src/server/http_server.cpp:2467">
P2: `launch_semantic_tool_prediction` shares one `NativeSemanticToolPredictor` instance (`native_semantic_predictor_`) across every request and invokes `native->predict(payload, tools)` from a fresh `std::async` thread per request. `predict` is non-const and the object owns a `Qwen3ToolPredictorIpcClient ipc_` with no visible synchronization; concurrent tool-using requests can therefore call `predict` concurrently on the same IPC client. If that client is not internally thread-safe, this is a data race. Additionally, spawning two `std::async(std::launch::async)` threads per tool request is unbounded under concurrency.</violation>

<violation number="2" location="server/src/server/http_server.cpp:2664">
P2: When upstream proxy mode is enabled with a before-model native predictor, proxied requests wait for an unused prediction before forwarding. Skip semantic prediction for forwarded requests, or explicitly cancel it before the early `forward_upstream()` return.</violation>

<violation number="3" location="server/src/server/http_server.cpp:4572">
P2: In overlap mode (the default for HTTP predictors, and the experimental native overlap schedule), `launch_semantic_tool_prediction` starts the predictor on a `std::async` worker before the request is enqueued, then `finish_tool_speculation` calls `req.automatic_tool_speculation.get()` / `req.semantic_tool_prediction.get()`. If generation finishes before the prediction (e.g. the model returns a plain answer with no tool call, or a short response), `.get()` blocks the worker thread for the remaining predictor time (up to `--tool-hint-timeout-ms`, default 2000 ms). This delays that response and, because the worker thread is shared, stalls other queued jobs. The overlap schedule is supposed to run generation and prediction concurrently, so blocking on the prediction here partially negates it.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/cpu_sparse_tool_executor.cpp">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/cpu_sparse_tool_executor.cpp:141">
P1: When an allowlisted call contains these extra fields, the executor can allocate several GiB despite the small public `iterations` schema. Reject unknown arguments and keep the qualified sparse shape fixed, rather than letting model-supplied JSON select dimensions and worker count.</violation>

<violation number="2" location="optimizations/ooo_spec_lucebox5_cpu/cpu_sparse_tool_executor.cpp:179">
P1: When this executor is configured without a CPU lane, the empty expected mask bypasses isolation and the workload competes with model decoding. Require a non-empty expected affinity and fail closed when it is absent.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/run_native_cpu_server_lucebox5.sh">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/run_native_cpu_server_lucebox5.sh:43">
P2: When `CANDIDATE_BUILD` is overridden for this launcher, the qualified launcher clears that ambient override before the wrapper runs, so the wrapper may execute its symlink/default build instead of the validated build. Pass build selection through a durable launcher-supported mechanism, or preflight the wrapper’s actual candidate.</violation>

<violation number="2" location="optimizations/ooo_spec_lucebox5_cpu/run_native_cpu_server_lucebox5.sh:49">
P2: When `PREDICTOR_TIMEOUT_MS` is overridden, this launcher drops the override before the qualified launcher starts the wrapper, so prediction continues using the 2000 ms default. Propagate the timeout setting here so it reaches `--tool-hint-timeout-ms`.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/bfcl_replay_tool_executor.py">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/bfcl_replay_tool_executor.py:53">
P2: When `cpu_affinity` is explicitly null or another falsy non-list, `or []` skips the affinity check and can report an unisolated executor as successful. Use a default-only lookup so malformed supplied values reach the type validation.</violation>
</file>

<file name="server/src/server/chat_template.cpp">

<violation number="1" location="server/src/server/chat_template.cpp:390">
P2: The new `tool_call_required` parameter is documented (header) as strengthening the template for OpenAI `tool_choice="required"` and forced-function requests, and both call sites in http_server.cpp pass `tool_choice_requires_call(req.tool_choice)`. However it is only honored in the DEEPSEEK4 case (line 390); the QWEN3, LAGUNA, and GEMMA4 branches ignore it entirely. For QWEN3 — the default/primary architecture for the workflows targeted by this PR — a `tool_choice="required"` or forced-function request produces the same prompt as a normal request, and its preamble even says "If there is no function call available, answer the question like normal", so the model is not actually forced to emit a call. Wire the flag into the other tool-capable branches (at minimum QWEN3, and LAGUNA's non-thinking/thinking tool-block) so the required-call contract holds across architectures.</violation>
</file>

<file name="server/src/qwen3/qwen3_loader.cpp">

<violation number="1" location="server/src/qwen3/qwen3_loader.cpp:150">
P1: When the compact GGUF has Q8_0 projection weights but BF16 embeddings or output weights, this assignment applies Q8_0 to every 2-D tensor and the drafter fails to load. Allocate each tensor using its source storage type or explicitly convert the non-Q8 tensors instead of deriving one global type from `blk.0.attn_q.weight`.</violation>

<violation number="2" location="server/src/qwen3/qwen3_loader.cpp:151">
P2: When an F16 GGUF is loaded on HIP, this branch accepts it but leaves `out.weight_type` as BF16, so the loader rejects the F16→BF16 copies later. Set `out.weight_type` to F16 for an F16 source or add an explicit F16→BF16 conversion.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/trace_compiled_tool_executor.py">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/trace_compiled_tool_executor.py:100">
P2: When a leaf executor returns a non-object envelope, `.get("ok")` raises an uncaught `AttributeError` and the child emits no protocol response. Check the envelope type before calling `.get()`.</violation>
</file>

<file name="server/src/common/qwen3_tool_predictor_ipc.cpp">

<violation number="1" location="server/src/common/qwen3_tool_predictor_ipc.cpp:169">
P1: When native prediction starts, the daemon inherits every non-CLOEXEC descriptor held by the server and target backend. That allows the predictor process to retain or inspect model, tool, or server IPC descriptors, so this path does not provide the promised descriptor privacy. Launch the predictor with a close-on-exec descriptor policy or an explicit inherited-descriptor allowlist before enabling this lane.</violation>
</file>

<file name="server/src/server/semantic_tool_hint.cpp">

<violation number="1" location="server/src/server/semantic_tool_hint.cpp:109">
P3: `parse_content_call` scans every byte of predictor `content`, and for each position holding `{` it runs a full `json::parse(content.begin()+offset, content.end(), nullptr, false)` over the rest of the string. This is O(n²) on unpredictable sidecar output and can be triggered on arbitrary predictor text (HTTP fallback path in `parse_semantic_tool_prediction`). It also accepts the first embedded JSON object as the tool call with no envelope/delimiter requirement, so a response whose real call is not the first object (or that has trailing prose after the object) silently falls through to the authoritative call, defeating the speculative fast path. The per-offset full-parse is the main cost; tail after the object also makes most scan positions fail.</violation>

<violation number="2" location="server/src/server/semantic_tool_hint.cpp:243">
P2: When a sidecar returns multiple `tool_calls` plus content containing one JSON call, this condition skips the array and accepts the content. Reject non-single `tool_calls` responses instead of selecting one representation, so malformed predictions cannot start speculation.</violation>

<violation number="3" location="server/src/server/semantic_tool_hint.cpp:377">
P2: When `tool_choice` is `"none"`, the native prompt still asks for functions, so it can launch an allowlisted speculative call that authoritative decoding will always cancel. Return an opt-out/error prompt for `none` before scheduling prediction.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/benchmark_trace_compiled_workflows.py">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/benchmark_trace_compiled_workflows.py:617">
P1: When `--tool-cpus` differs from the server configuration, the benchmark compares different CPU lanes and can run authoritative tools on model CPUs. Require `tool_speculation.tool_cpu_affinity == args.tool_cpus` before measuring.</violation>

<violation number="2" location="optimizations/ooo_spec_lucebox5_cpu/benchmark_trace_compiled_workflows.py:977">
P1: The final-answer gate never consumes the generated tool result because `post_final()` is context-free and receives the expected receipt directly. Send the accumulated assistant/tool conversation to the final turn and validate that response instead.</violation>

<violation number="3" location="optimizations/ooo_spec_lucebox5_cpu/benchmark_trace_compiled_workflows.py:1453">
P2: When either trace path is customized, the benchmark and executor read different files because the executor paths are never propagated. Require matching executor environment at launch or reject non-default paths before running.</violation>
</file>

<file name="server/src/common/backend_ipc.cpp">

<violation number="1" location="server/src/common/backend_ipc.cpp:452">
P2: The new ownership/mode and lstat checks are applied in the shared `BackendIpcProcess::init_work_dir`, so they tighten every backend-IPC mode (remote DFlash draft, PFlash compress, target shards, moe-expert-compute), not just the new Qwen3 predictor. Any existing deployment that passes a user-supplied `--remote-*-work-dir` that is a symlink (previously accepted via `stat` + `S_ISDIR`) or is owned by a different euid / has mode other than exactly 0700 (e.g. a shared scratch dir) now hard-fails `BackendIpcProcess::start` at startup. If this hardening is intended only for the predictor lane, scope it to that caller or document the cross-mode behavior change for existing multi-GPU deployments that already configure `work_dir`.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/dflash_server_native_tool_predictor_wrapper.sh">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/dflash_server_native_tool_predictor_wrapper.sh:29">
P2: When an operator sets the valid cache-disable value `PREFIX_CACHE_SLOTS_OVERRIDE=0`, this wrapper rejects it before starting the server. Accept zero in the override regex so the wrapper can pass `--prefix-cache-slots 0`.</violation>

<violation number="2" location="optimizations/ooo_spec_lucebox5_cpu/dflash_server_native_tool_predictor_wrapper.sh:45">
P2: When a binary override exists but is not an executable regular file, this preflight accepts it and startup fails later with a generic exec/IPC error. Check the two binaries with `-f && -x` and the model with `-f` before launching.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/README.md">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/README.md:88">
P3: The override advice here conflicts with the next sentence. This paragraph tells operators to override placement with `PREDICTOR_MODEL`/`PREDICTOR_GPU`/`PREDICTOR_MAX_CTX`/`PREDICTOR_MAX_TOKENS`/`PREDICTOR_TIMEOUT_MS` env vars, but immediately notes the qualified launcher clears ambient variables (which is why the `candidate-build` symlink is needed as the durable override). If that launcher clears ambient env, those `PREDICTOR_*` overrides are discarded when launching via `run_native_cpu_server_lucebox5.sh`, so the documented override silently does not take effect through the qualified path. Clarify that the `PREDICTOR_*` overrides apply when launching the wrapper directly, or state how to pass them so they survive the launcher.</violation>
</file>

<file name="server/test/smoke_qwen3_tool_predictor_ipc.cpp">

<violation number="1" location="server/test/smoke_qwen3_tool_predictor_ipc.cpp:190">
P3: The batch gate returns success when names match but does not require exact argument matches, even though exact_matches is already tracked and reported. With the current baseline at 9/12 exact args, this automation would still exit 0 on an argument-canonicalization regression, which is exactly the safety property this PR emphasizes. Gate on exact_matches == cases.size() (or assert the expected baseline explicitly) so the check protects the exact-match commit guarantee.</violation>
</file>

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/server/chat_template.cpp">

<violation number="1" location="server/src/server/chat_template.cpp:390">
P2: The new `tool_call_required` parameter is documented (header) as strengthening the template for OpenAI `tool_choice="required"` and forced-function requests, and both call sites in http_server.cpp pass `tool_choice_requires_call(req.tool_choice)`. However it is only honored in the DEEPSEEK4 case (line 390); the QWEN3, LAGUNA, and GEMMA4 branches ignore it entirely. For QWEN3 — the default/primary architecture for the workflows targeted by this PR — a `tool_choice="required"` or forced-function request produces the same prompt as a normal request, and its preamble even says "If there is no function call available, answer the question like normal", so the model is not actually forced to emit a call. Wire the flag into the other tool-capable branches (at minimum QWEN3, and LAGUNA's non-thinking/thinking tool-block) so the required-call contract holds across architectures.</violation>
</file>

<file name="server/src/server/tool_speculation.cpp">

<violation number="1" location="server/src/server/tool_speculation.cpp:779">
P2: The FD-isolation guarantee is only enforced on glibc >= 2.34. `addclosefrom_np` is guarded by `__GLIBC_PREREQ(2, 34)`, so on glibc 2.17-2.33, musl-based Linux, or any non-glibc POSIX build the child executor inherits every open descriptor from the long-running server (listening sockets, live client connections, model IPC pipes, accelerator descriptors) — directly contradicting the header comment "The executor receives only stdin/stdout/stderr" and the PR's privacy claim. The failure is silent: the code compiles and launches without any FD isolation.</violation>
</file>

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/server/http_server.h">

<violation number="1" location="server/src/server/http_server.h:480">
P1: When an allowlisted tool legitimately returns JSON `null`, dependent calls are skipped as failed. Track result availability separately from the JSON value instead of using `is_null()` as the success sentinel.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/realapi/dag_bench.py">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/realapi/dag_bench.py:81">
P2: When a DAG references a nested result field such as `$1.meta.latitude`, `resolve_args` does not resolve the path. Parse the complete dotted path and traverse the result value for both whole and embedded placeholders before executing the dependent call.</violation>

<violation number="2" location="optimizations/ooo_spec_lucebox5_cpu/realapi/dag_bench.py:270">
P2: When running `d09`, a valid summary-only answer with no digits is marked incorrect, and the completion heuristic may spend up to four extra model turns nudging it for numbers. Make completion and correctness validation task-specific instead of unconditionally requiring a digit.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/realapi/rest_tools.py">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/realapi/rest_tools.py:84">
P2: When generation crosses a weather or rate update, an exact name-and-arguments match commits this earlier live snapshot as the current result. Add a freshness/age check before committing these tools, or exclude time-varying tools from speculative hits.</violation>

<violation number="2" location="optimizations/ooo_spec_lucebox5_cpu/realapi/rest_tools.py:124">
P1: When a REST request fails, this line returns `ok: false` inside the raw result, but the executor wraps every raw result in an outer `ok: true`; the server therefore reports a speculative hit for an error. Propagate raw failures as executor failures, or have the executor reject results whose `result.ok` is false, so clients retry through the authoritative path.</violation>
</file>

<file name="server/src/deepseek4/deepseek4_backend.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_backend.cpp:2270">
P2: When a stale snapshot is restored with a prompt ending exactly at `snap_pos`, generation skips prefill and fails with `DecodeFailed` because `last_logits_` is empty. Reject stale slots for full-length restores or fall back to a fresh prefill before decoding.</violation>
</file>

<file name="server/src/server/http_server.cpp">

<violation number="1" location="server/src/server/http_server.cpp:4219">
P2: When an earlier wrapper is unparseable or contains multiple calls, this shifts later `$k.path` references and can reject a valid dependent call as `unresolvable_dependency`. Track a separate 1-based parsed-call index and map it to the corresponding entry.</violation>

<violation number="2" location="server/src/server/http_server.cpp:4222">
P2: When two authoritative calls have the same canonical name and arguments but different IDs, this marks the second as `duplicate_call` and never launches it. Keep one attempt per authoritative call and match each result to its corresponding ID instead of deduplicating by arguments.</violation>
</file>

<file name="server/src/server/tool_speculation.h">

<violation number="1" location="server/src/server/tool_speculation.h:148">
P1: When a client supplies `tool_speculation.confidence: 1.0` without a resource profile, this new `enabled()` condition admits it and `ToolSpeculationAttempt` launches it before target generation. Do not use confidence as provenance; distinguish request predictions from model-emitted early-dispatch calls before bypassing the profile gate.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/realapi/rest_summary.py">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/realapi/rest_summary.py:15">
P3: The per-task ratio (c["wall_ms"] / s["wall_ms"]) and the aggregate (ct/st) divide by spec wall time with no zero guard. A spec run recording 0 wall_ms aborts the whole summary. Guard before dividing.</violation>

<violation number="2" location="optimizations/ooo_spec_lucebox5_cpu/realapi/rest_summary.py:19">
P3: statistics.median(ratios) and statistics.mean(ratios) raise StatisticsError when ratios is empty, which happens whenever no task has both a control and a spec arm. Guard for the empty case before computing median/mean.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/realapi/rest_bench.py">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/realapi/rest_bench.py:126">
P2: When the model emits a supported `$k.path` dependent call, this map keys the hit by resolved-argument digest while the client hashes raw placeholders, so it misses the completed result and executes an invalid client call. Key early hits by the authoritative `call_id` or by the entry's raw `name` and `arguments` template.</violation>

<violation number="2" location="optimizations/ooo_spec_lucebox5_cpu/realapi/rest_bench.py:151">
P2: Because `expect` contains words copied from each prompt, a model can answer without any tool call and still be counted correct, invalidating the accuracy and control/spec comparison. Require successful tool execution and validate returned values rather than checking prompt keywords only.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/realapi/launch_server.sh">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/realapi/launch_server.sh:20">
P2: This script checks only `pgrep -x dflash_server` before launching, so it will happily start over a GPU already owned by any other workload. The mirrored `run_native_cpu_server_lucebox5.sh` refuses when `/dev/kfd` has an owner (`fuser -s /dev/kfd` → exit 75). Add the `/dev/kfd` guard to avoid corrupting or crashing an in-use GPU card.</violation>

<violation number="2" location="optimizations/ooo_spec_lucebox5_cpu/realapi/launch_server.sh:36">
P2: The script's header advertises the native Qwen3-0.6B predictor and even validates predictor_model, but its `exec env -i` block never sets `PREDICTOR`. Because `env -i` clears the environment and the wrapper defaults `PREDICTOR=0`, the predictor lane stays silently off. The mirrored `run_native_cpu_server_lucebox5.sh` sets `PREDICTOR="1"` here; add it so the advertised predictor is actually enabled.</violation>
</file>

<file name="server/src/server/tool_speculation.cpp">

<violation number="1" location="server/src/server/tool_speculation.cpp:996">
P2: When `--tool-spec-timeout-ms` is near `INT_MAX`, this multiplication overflows before the duration is constructed, producing an invalid absolute deadline and causing executor results to time out immediately. Convert to a wider duration before multiplying or cap the accepted timeout.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/realapi/results/dag_small_4tasks_3arms.json">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/realapi/results/dag_small_4tasks_3arms.json:1267">
P2: The committed s04 `dag` result's final answer is corrupted with U+FFFD replacement characters where digits were dropped ("7���2", "8���0", "1���5th-century", "1���3.9 km/h"), while the same task's `parallel` arm produced the clean text ("2,430 meters", "13.9 km/h"). Only the dag/early-dispatch run is garbled, which contradicts the PR's "lossless" claim for tool-result passthrough. Confirm this is an external-API byte anomaly and not the tool-result encoding being lossy-decoded in the early-dispatch/dependency lane (bytes replaced with U+FFFD), then re-run; a degraded result should not stand as evidence of correctness.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/realapi/restart_server.sh">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/realapi/restart_server.sh:6">
P2: When the `logs/` directory does not exist under the cd'd path, the `> "$log"` redirect fails silently and the server output is discarded. The success grep and the `tail -20 "$log"` error path then read a nonexistent file, so the loop never sees "listening on" and the script burns the full 240x5s window before reporting a misleading `TIMEOUT`. Create the directory before writing the log so the failure is detected at launch.</violation>
</file>

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="optimizations/ooo_spec_lucebox5_cpu/README.md">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/README.md:76">
P2: Following the Reproduce instructions never applies the intended executor or allowlist. launch_server.sh reads TBSPEC_EXECUTOR and TBSPEC_ALLOW (defaulting to the terminal-bench tb_tool_executor.py and `read_file,list_dir,search_files`), and its `env -i` discards the TOOL_SPEC_EXECUTOR/TOOL_SPEC_ALLOW and BUILD_DIR=$PWD/realapi you pass. As written, the server launches with the wrong tools, so rest_bench/dag_bench calls are not allowlisted and the documented measurements cannot be reproduced. Set TBSPEC_EXECUTOR and TBSPEC_ALLOW instead of TOOL_SPEC_*, and drop the ineffective BUILD_DIR.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/realapi/wrapper_dflash_server.sh">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/realapi/wrapper_dflash_server.sh:42">
P2: When an operator sets `LUCE_MMVQ_MAX_NCOLS`, this wrapper discards the explicit dispatch override and forces the q=5 threshold. Preserve the caller's value while using 5 only as the default.</violation>
</file>

<file name="server/src/common/backend_ipc.cpp">

<violation number="1">
P2: When a signal interrupts shutdown, `close()` forgets the still-running child and can leave a zombie. Retry `waitpid` on `EINTR` before clearing `pid_`.</violation>

<violation number="2">
P2: When work-directory creation succeeds but pipe setup or required shared transport validation fails, these early returns leave the owned temporary directory behind. Call `close()` or explicitly remove the owned directory on every post-initialization failure path.</violation>

<violation number="3">
P1: If the daemon stalls before sending its readiness status, `start()` blocks indefinitely on this read and prevents the server from falling back or reporting startup failure. Restore a bounded readiness deadline and terminate the child when it expires.</violation>
</file>

<file name="server/src/server/http_server.cpp">

<violation number="1" location="server/src/server/http_server.cpp:904">
P3: The `/props` predictor metadata now contains unreachable constant branches, which makes the advertised capability contract difficult to audit. Replace these expressions with direct `false`/`json(nullptr)` values or remove the obsolete fields.</violation>

<violation number="2" location="server/src/server/http_server.cpp:3979">
P2: When an early-dispatch response contains a dependent call, `prefetch_next_turn` always returns before prefill because dependent hits never receive `tool_message_content`. Add the same canonical-content assignment in the dependent-hit branch.</violation>

<violation number="3" location="server/src/server/http_server.cpp:4008">
P2: When `prepare_inline_snap` selects the lookup's `restore_slot`, this line frees the source snapshot before prefetch restores from it. Cancel the reservation and skip prefetch when `snap_slot == restore_slot` so the valid end-of-turn cache entry is preserved.</violation>
</file>

<file name="server/src/qwen3/qwen3_backend.cpp">

<violation number="1">
P1: Writing the decoded token one slot too far: in do_decode, `committed` is incremented before this call (see `committed++; cache_.cur_pos = committed;` above), so the just-sampled `next` belongs at zero-based position `committed - 1`, not `committed`. This leaves a hole in the KV cache at the first decoded slot and shifts every subsequent decoded token's K/V one position past its true position (do_step writes K/V at `kv_start` via the cache view and assigns rope positions from `kv_start+i`). The prior `committed - 1` was correct and matches the sibling call sites that write `first` at pre-increment `cur_committed`. Restore `committed - 1` so generation (and any end-of-turn snapshot) reflects the real token sequence.</violation>
</file>

<file name="server/src/server/server_main.cpp">

<violation number="1" location="server/src/server/server_main.cpp:634">
P2: When `--prefetch-prefill` is passed without `--early-dispatch`, startup succeeds but every prefetch is skipped because `output.early_dispatch` remains false. Reject this incomplete flag combination after parsing, including the documented `--end-turn-snapshot` prerequisite, instead of silently disabling the requested optimization.</violation>
</file>

<file name="optimizations/ooo_spec_lucebox5_cpu/realapi/rest_bench.py">

<violation number="1" location="optimizations/ooo_spec_lucebox5_cpu/realapi/rest_bench.py:151">
P3: This change makes the tool-message text differ between the two benchmark arms, contradicting the script's stated design that "the ONLY difference is the request field automatic_tool_speculation." In the spec arm, early-hit tool results are echoed as the server's canonical text (rapidjson key ordering via tool_message_content) while non-hit calls and all control-arm calls use the client's Python json.dumps formatting. The values are identical (a hit is an exact canonical match), so timings are unaffected, but the correctness comparison feeds the model differently formatted tool text in each arm and can confound the paired result. If consistency matters, dump the value with the same compaction/sorting in both arms, or document the intended divergence.</violation>
</file>

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/server/server_main.cpp">

<violation number="1" location="server/src/server/server_main.cpp:222">
P3: The CLI validates `--tool-spec-max-executors` as `>= 0`, and the executor lane treats 0 as "no cap" (tool_speculation.cpp only enforces the cap when `max_concurrent_executors > 0`). The usage text documents only the default (16) and never says that 0 disables the cap, so an operator can pass 0 expecting zero executors and instead get unlimited concurrency. Document that 0 = unlimited in the flag help.</violation>
</file>

<file name="server/src/server/http_server.cpp">

<violation number="1" location="server/src/server/http_server.cpp:4043">
P3: When a queued request cancels prefill, the backend records cancellation on its copied `DaemonIO`, so `prefetch_io.cancelled` remains false and this branch never logs that prefetch yielded. Check the queue state or propagate cancellation through shared state instead.</violation>
</file>

<file name="server/src/server/tool_speculation.cpp">

<violation number="1" location="server/src/server/tool_speculation.cpp:693">
P2: When `terminate_executor(true)` reaps a child during either grace wait, it returns before releasing the newly acquired executor slot. Subsequent dependency waves can therefore receive `executor_saturated` even though no child is running; release the slot on every terminal return, including the analogous wait-error path in `collect_executor_result`.</violation>
</file>